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In this article , the performance achieved by some specific data compression al- 
gorithms is compared with absolute limits prescribed by rate distortion theory for 
Gaussian sources under the mean square error distortion criterion. These results 
show the gains available from source coding and can be used as a reference for the 
evaluation of future compression schemes. Some current schemes perform well, but 
there is still room for improvement. 


I. Introduction 

The theoretical limits on the performance of source and 
channel coding are well known for several source and chan- 
nel models [1,2,5], In this article, the authors calculate 
the theoretical limits for one- and two-dimensional Gauss- 
Markov sources used as models for planetary images. The 
formulas underlying these calculations are well known; the 
aims in this article are first to collect and graphically dis- 
play these results, and then to compare them with the 
performance of specific data compression algorithms. 

These results show the gains available from source cod- 
ing and can be used as a reference for the evaluation of 
present and future compression schemes. These results 
also suggest that large improvements in information trans- 
mission in future missions can be achieved by advanced 
source coding. 

II. Theoretical Rate Distortion Limits 

The authors consider time-discrete continuous- 
amplitude sources that produce identically distributed 


output samples x governed by a probability distribution 
P(x) with density p(x). Each source sample x is recon- 
structed after source coding and decoding into a recon- 
structed sample y. The accuracy of reproduction is mea- 
sured by a nonnegative function d(x, y) — (x — y) 2 called a 
squared error distortion measure. The average^ distortion 
D on a sequence of N samples is (1/N)X^_ 0 ( x * — y*) 2 
and is called mean square error (MSE) distortion. 

A. One-Dimensional Gaussian Sources 

For a Gaussian memory less source, p(x) is the Gaussian 
probability density with variance cr£, and the rate distor- 
tion function for MSE distortion is [1] 

R(D) = \ log 2 ^, o <D<*1 (1) 

where the rate R is measured in bits/sample. 

A time-discrete stationary Gaussian source with spec- 
tral density function 
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$(w) = 


fc = -oo 


( 2 ) 


where <j>(n) is the autocorrelation function, has a rate dis- 
tortion R(D) given in the parametric form [1] 


and 


i r 

D(9) = — J min[0,$(w)]du> 


(3) 




max 


„ 1, ^(w) 1 

O.jloS,— 


du 


(4) 


where 0 is the parameter. 

Consider the special case of a first-order Gauss-Markov 
source of variance cr\ with samples 


Xi - pxi-i + Wi 


(5) 


m = (^) 2 [o, i io g2 $(u, g — ] ^ lW2 

(9) 

A two-dimensional Gauss-Markov (autoregressive) causal 
source is defined by 


Xi J = PlZi-lj + P 2 Xi,j-l + Pl,3*i-U-1 + w >'./ ( 10 ) 

where {ivjj} is a two-dimensional i.i.d. zero-mean Gaus- 
sian sequence with variance <7^, . If Pi , 2 — —P 1 P 2 is chosen, 
the source model in Eq. (10) becomes separable and will 
be called the two-dimensional causal (2DC) model. Then 
the variances of the sequences and {r, i2 } are related 

by crl = <r*(l - />?)( 1 - Pi), and 

, *2(1 -/>?)(! -Pi) 

(wi,w 2 ) (1 _ C oswi + Pi)(l — 2p 2 cosw 2 4- p|) 

( 11 ) 


where {«>»} is an independent, identically distributed 
(i.i.d.) zero-mean Gaussian sequence with variance #1 = 
-p*). This source will be called the one-dimensional 
causal model, or 1DC model, and is characterized by an 
exponentially decaying memory given by the autocorrela- 
tion function 


= <r 2 x \p\ n , 0<p<l (6) 


which gives 


*(w) = 


*2(i - p 2 ) 

1 — 2pcosu; + p 2 


(?) 


Incidentally, the power spectral density function is always 
easy to find, given the definition of the model that gener- 
ates the samples {a:*}, as described in [3]. 

B. Two-Dimensional Gaussian Sources 

The rate distortion function R{D) for a two- 
dimensional Gaussian source is given by [7] 

D{Q) = ( — ) 2 / f min[^,$(uii,W2)]dwidui2 (8) 

27T J-x 


This causal separable model has an autocorrelation func- 
tion 0(ni,n 3 ) given by 

0(ni,n 2 ) = al\ Pi r\ P2 r ( 12 ) 


which displays an undesirable nonisotropic behavior, as 
discussed later in this section. 

Figure 1 shows the rate distortion functions for the 1DC 
model and the 2DC model with p\ = P 2 = P for several 
values of p. The values of the correlation coefficient p 
have been chosen to illustrate the effect of correlation on 
the rate necessary to represent the source. At low distor- 
tion, these values give rate distortion curves spaced by an 
integer number of bits from the curve for the memoryless 
source. Each successive correlation value in Fig. 1 repre- 
sents (asymptotically for low distortion) one extra bit of 
information that can be extracted from each sample’s cor- 
relation with its neighbors, and thus need not be spent to 
represent the source. 

A more realistic model for images, the two-dimensional 
noncausal (2DNC) model, is given by 

Xij = a(x<j_i + x itj +i + Xi-u + *f+i,;) + u>«\; , 

|a| < 1/4 (13) 
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This is a noncausal model with a power spectral density 


$(wi,w 2 ) _ 2a(coswi + cos wo)] 2 


To (14) 


where <r 2 = c 2 »?, and 


-i ( _1_\ f f - r~du>idu2 

1 \2tt ) J-xJ-tl 1 - 2a(cosw! + cosw 2 )]- 

(15) 


The autocorrelation of this model was computed numeri- 
cally, and it was found to behave almost isotropically at 
small displacements. Figure 2 shows a comparison of the 
autocorrelations for the causal and noncausal models for 
correlation coefficients p that are representative of typical 
planetary images. The function 

4>(nx,n 2 ) = 4\p\^T^i (16) 


is an example of exactly isotropic autocorrelation [4], but 
the authors do not presently know a model that realizes 
such an autocorrelation. 

The qualitative behavior of the autocorrelation func- 
tions for the 2DC and 2DNC models is illustrated in the 
contour plots of Fig. 3. Note that for small values of n x 
and n 2 the contours for the 2DNC model are nearly circu- 
lar, indicating that this model is nearly isotropic for small 
displacements. 

The rate distortion functions for the 2DC and 2DNC 
models are shown in Fig. 4 with the same parameter values 
used in Fig. 2. Since the autocorrelation function for the 
2DNC model decays more rapidly than for the 2DC model 
when both models have the same value of <^>(1,0) fixed, the 
rate distortion function of the 2DNC model lies above that 
of the 2DC model. 


{7\, •••, 7m_ i}, the output is y = Lk if and only if 
7jt-i < x < T*. k = 1, •••, M, where T 0 = -oo and 
Tjy = -{-oo. The input-output characteristic of a four- 
level quantizer is shown in Fig. 5. 

Let {*,, •••> xjv} be a sequence of random samples 
generated by a source and let {t/i, • •' , !/jv} be tbe corre " 
sponding quantized samples produced by an M-level quan- 
tizer. Then the quantized sequence has rate B = log 2 M 
bits and MSE distortion 


D i i ^ £[(*,- - y») 2 ] = £[(*< - Vi) 2 ] 

» = 1 

M r T h 

= £ / ( x ~ L kfv{x)dx (17) 

Jb=sl '* Tk - x 


where p(x) is the probability density of the source. There- 
fore, the M-level quantizer realizes the point (B,D) on 
the rate distortion plane. The optimum quantizer, which 
achieves the lowest possible MSE for given source statis- 
tics, has been determined in terms of the reproduction 
levels {£,*} and the thresholds {T k } using an optimization 
technique developed by Lloyd and Max in 1960. If the 
quantizer is restricted to have equally spaced thresholds, 
he., a uniform quantizer with constant step size T k — T k -i 
is considered, a slightly higher distortion for correspond- 
ing rates is obtained, as shown in Fig. 6 for the Gaussian 
memoryless source. An optimum uniform quantizer is a 
uniform quantizer that minimizes the MSE distortion. 

Improved rate performance can be obtained by using 
entropy coding after quantization, since the probability 
P k s s Pr(y = L h ) = p(x)dx that a quantizer output 

will be L k is not a constant (except for degenerate cases), 
and therefore the entropy of the quantized samples y is 
strictly less than B 


N 

H(y) = ~Y, Pk lo S2 Pk < B 

Jk=l 


(18) 


III. Quantization 

Given a time-discrete continuous amplitude source, the 
simplest form of data compression is scalar (sample-by- 
sample) quantization. An Af-level quantizer is a device 
with an input that can assume any real value x and an 
output y that can assume only M values {Li, ■ • • , Lm}- 
Usually, the number of levels is a power of 2, so that a B-bit 
quantizer has 2 B levels. Given the quantization thresholds 


The entropy coded performance of the two quantizers con- 
sidered above is also shown in Fig. 6, where it is apparent 
that the advantage of the Lloyd-Max quantizer over the 
uniform quantizer disappears after entropy coding. Re- 
sults on entropy coded quantizers were obtained from the 
literature [3] and reproduced by computer simulation. 

Instead of quantizing individual source samples, one 
could collect a whole vector x — (xj, •••, x n ) and then 
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vector quantization. The performance of vector quantiza- 
tion methods will be discussed in a future article. 

If one replaces the memoryless Gaussian source with 
a one-dimensional Gauss-Markov source with correlation 
coefficient p between successive samples (1DC model), a 
simple method to exploit the source memory is to take 
differences between successive quantized samples and then 
apply entropy coding. The performance of such a one-step 
predictor on samples produced by an optimum uniform 
quantizer is shown in Fig. 7. 

In practice, the continuous amplitude source is initially 
quantized to B bits, typically 8 bits. In the following dis- 
cussion of practical compression algorithms for images, it 
is assumed that the source has been quantized to 8 bits 
per sample by an optimum uniform quantizer. 

IV. Comparisons of Practical Compression 
Algorithms and Theoretical Limits 

The performance of specific compression algorithms de- 
signed for 8-bit input data can be measured experimentally 
by generating in software a Gauss-Markov random field ac- 
cording to one of the models described in Section II and by 
quantizing the resulting samples to 8 bits with an optimum 
uniform quantizer. 

The entropy coded one-step predictor described in the 
previous section is a simple example of a practical com- 
pression scheme, and it is essentially the image compres- 
sion scheme used in Voyager, where the source was initially 
quantized to 8 bits by the camera. The point denoted by 8 
in the rate distortion plot of Fig. 7 represents the so-called 
lossless performance of such a scheme. This scheme per- 
forms reasonably well at low distortions (as compared with 
the rate distortion function) when it is applied to the one- 
dimensional source 1DC. One will see that its performance 
is no longer attractive when applied to two-dimensional 
sources 2DC or 2DNC. 


The proposed Joint Photographic Expert Group 
(JPEG) image compression standard [6] uses, in its base- 
line version, discrete cosine transform (DCT) processing, 
quantization, and Huffman coding. The performance of 
this compression scheme on the 2DC model has been evalu- 
ated and compared with the rate distortion limits in Fig. 8. 
The performance of the entropy coded one-step predictor 
on the 2DC model is also shown in Fig. 8 for compari- 
son. For most science purposes, a typical planetary image 
is considered acceptable at normalized distortions D/<r x 
up to approximately 10~ 2 , corresponding to about 5 gray 
levels of rms error out of 256 levels for typical images. In 
this range of interest, the JPEG scheme is superior to the 
entropy coded predictor, but the theoretical limit leaves 
ample space for improvements. The performances of the 
JPEG scheme and the entropy coded one-step predictor 
on the 2DNC model are compared in Fig. 9. 


V. Conclusion 

The theoretical limits computed in this article and the 
experimental results on source models verify the gains 
available by source coding, and can be used as a refer- 
ence for the evaluation of present and future compression 
schemes. These results also suggest that large improve- 
ments in information transmission in future missions can 
be achieved by advanced source coding. 

Mathematical source models studied in this article in- 
clude both relatively simple one- and two-dimensional 
causal Gauss-Markov models and a twodimensional non- 
causal model whose nearly isotropic correlation function 
more closely resembles that of real images. 

More work is necessary in relating the mathematical 
models to actual image sources, in evaluating the per- 
formance of other practical compression schemes, and 
in understanding the actual quantization performed in 
the camera. 
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Fig. 1. Rat* distortion functions for 1DC and 2DC models. 
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Fig. 2. Two-dimensional normalized autocorrelation functions 
0(^1, "2)/<r* : ( a ) 2DC model and (b) 2DNC model. 
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Fig. 3. Contour plots*, (a) causal and (b) noncausal autocorrelations. 
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Fig. 4. Rato distortion functions for causal and noncausal two- 
dimensional models. 
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Fig. 6, Performance of quantization schemes for memoryless 
Gaussian sources. 


Fig. 5. Input-output characteristic of a four-level quantizer. 
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